Introduction

The semantic index is a set of stratego libraries, java primitives and editor extensions that provide a language-parametric framework for doing name binding, scoping, incremental analysis and incremental compilation.

Quick start

To be able to use the index a nightly version of Spoofax is required which can be retrieved from the following update site:

   http://spoofax.org/update/nightly

The simplest place to get an example on what the index can do and how a project using the index looks like is generating a new Spoofax project. Go to File->New->Project, choose "Spoofax editor project" and fill in Entity as the language name. This will generate and build a Spoofax example project for you that uses the index. To test it out open up test/example.ent and try some things like resolving a reference, code completion and introducing errors by changing types to types that do not exist. These features are all provided by the index using naming annotations in the SDF file and Stratego code to fill in some gaps in the generic algorithm. The tutorial will go into more detail on how to define these things yourself.

More complex examples using the index are also available. We've prototyped a mini C# subset, a C subset without macros and the entity language with functions and aspects using the index. To get these examples download this archive^?, extract it somewhere, import the projects into an Eclipse workspace and build all projects (ctr+b).

Tutorial

In this tutorial we will explain how the example entity language uses the index and how to extend it with functions and simple expressions.

Analysis

Name annotations

If you look at the entity language SDF definition below you will see that it has naming annotations on certain terminals (Type@=, Type@, etc) and scope annotations on productions (scope(Type)).

    "module" Module@=ID Definition*       -> Start {"Module", scope(Type)}
    "entity" Type@=ID "{" Property* "}"   -> Definition {"Entity", scope(Property)}
    Property@=ID ":" Type                 -> Property {"Property"}
    Type@ID                               -> Type {"Type"}

Adjust rules

Error checking

Incremental analysis

Compilation

Inner workings

Extension points

FAQ

Where can I find API documentation?

How to fix build error x?

Web	TWiki Site Map	Use to...
Main	The Main web is dedicated to the maintenance of this website. This is the place to discuss meta-issues such as what style to use, how best to organize a survey, what the ideal topic size is, how to refer to papers, what the preferred layout of the site should be, etc.	...
TWiki	Welcome, Registration, and other StartingPoints; TWiki history & Wiki style; All the docs...	...discover TWiki details, and how to start your own site.
Gmt	Generative Model Transformer
Gpce
Octave	The Stratego web is the home of Stratego, a language for program transformation based on the paradigm of rewriting strategies. The aim of this language is to provide an expressive and declarative language for expressing all kinds of program transformations. The web includes publications on Stratego, download of the StrategoCompiler, documentation, and descriptions of applications.	...
Sandbox	Sandbox test area with all features enabled.	...experiment in an unrestricted hands-on web.
Sdf	The Sdf web is dedicated to the modular syntax definition formalism SDF. Here you can find implementations, pointers to download pages, syntax definitions for common languages, discussions about new features and implementatios of SDF, and tips and tricks for using the formalism.	...
SdfBackup	The Sdf web is dedicated to the modular syntax definition formalism SDF. Here you can find implementations, pointers to download pages, syntax definitions for common languages, discussions about new features and implementatios of SDF, and tips and tricks for using the formalism.	...
Stratego	The Stratego web is the home of Stratego, a language for program transformation based on the paradigm of rewriting strategies. The aim of this language is to provide an expressive and declarative language for expressing all kinds of program transformations. The web includes publications on Stratego, download of the StrategoCompiler, documentation, and descriptions of applications.	...
Sts	The Sofware Transformation Systems wiki
Tiger	Home of the Tiger in Stratego project, which is concerned with the exploration of transformation techniques in compilation using an implementation of a Tiger compiler.	...
Tools	The Tools web is the home of the XT? bundle of program transformation tools. XT is an open framework for program transformation based on the ATerm format for exchange of programs between tools. The bundle includes packages for parsing, pretty-printing, term rewriting, and grammar recovery. It also contains a distribution of the SDF2? GrammarBase?. The OnlinePackageBase is an open collection of packages for program transformation supporting package bundling on demand.	...
Transform	The Transform web is an attempt to get an overview of program transformation research and application. In the first place the web is a collection of resources such as pointers to researchers, conferences, journals, summaries and reviews of papers, and tools for implementing transformation systems. In the second place the Transform web attempts to bring structure in the world of program transformation by means of categories, taxonomies such as the TransformationTaxonomy, and entry points such as the ReengineeringWiki and the DeCompilation page.	...
You can use color coding by web for identification and reference. This table is updated automatically based on WebPreferences settings of the individual webs. Contact webmaster@strategoxt.org if you need a separate collaboration web for your team. See also AdminTools.

Legend of icons:

= Go to the home of the web = Search the web

= See recent changes in the web = Subscribe to get notified of changes by e-mail

SpoofaxLang is a meta-language that can used to define new languages. It is currently an on-going project at the Delft University of Technology, The Netherlands.

Primary vision of developing this language to define and reuse software languages in a modular and reusable manner to support Software Language Composition. It is one of most the desirable features of Software Language Engineering to enable reusability of the existing language constructs.

"From now on, a main goal in designing a language should be to plan for growth." – Guy Steele: Growing a Language, OOPSLA’98 invited talk.

A language should be designed in such a way that evolving it with respect to different language constructs requires the minimum effort. It is well-known that designing a high quality language requires considerable effort. SpoofaxLang promotes languages as a reusable library with the vision of agile language development. Therefore, the cost of full-blown language can be reduced considerably.

Moreover, providing rich IDE support in language development is another area that will be addressed in this language. SpoofaxLang will be integrated with Spoofax - a language workbench to support various IDE features such as, code highlighting, outliner, various coloring scheme , reference resolving and so on.

Therefore, SpoofaxLang is yet another effort to realize language-oriented programming in a pragmatic way. It can be considered as a mean to achieve modular software language composition.

-- TobiVollebregt - 24 Mar 2012

Spoofax 1.1 -- released February 15, 2013

* NaBL name binding language

* Q&A on YellowGrass^?

Download

• Update sites changed:
  • http://download.spoofax.org/update/stable for STABLE release
  • http://download.spoofax.org/update/unstable for UNSTABLE build (currently out of date)
  • http://download.spoofax.org/update/nightly for TRUNK-based build

Documentation

License

Spoofax is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

Support

Despite the disclaimer above we do our best to help users of Spoofax.

Subscribe to the Stratego mailing lists^?, in particular the stratego-announce and stratego mailing lists to get announcements of new releases and ask questions about usage of the languages and tools. Also we're interested to know what people are using Stratego/XT for and how it might be improved, so feel free to drop us^? a line.

---

Spoofax 1.1 - released February 15, 2013

---

Summary of Changes

Full issue list on YellowGrass http://yellowgrass.org/tag/Spoofax/1.1

Publications

NaBL Name Binding Language

A major (we think it is revolutionary) innovation in Spoofax 1.1 is the NaBL name binding language.

Eclipse

• full support on Eclipse 3.7 and 3.8

• Spoofax should be used with Eclipse Classic, as it conflicts with Web Tools Platform included in Eclipse for Java Developers. This is detailed in the issue http://yellowgrass.org/issue/Spoofax/620 and there's a note about this at http://strategoxt.org/Spoofax/Download under Incompatibilities.

Project Building

• Generation of .gitignore file for new projects. Documented at http://webdsl.org/spoofaxwiki/wiki/version%20control

• A proper clean task in generated Ant builder, makes it easy to do a thorough clean job before committing to VC. Also documented at http://webdsl.org/spoofaxwiki/wiki/version%20control

• Project > Clean menu now cleans the project

• Much more of the generated files can now be safely deleted (clean takes care of this)

• The New Project wizard now supports creation of minimal project instead of full Entity

• Display warning on start-up if Eclipse is not started with the necessary arguments (stack, heap, permgen)

• Fix issue causing the file associations in Eclipse preferences to be unusable (http://yellowgrass.org/issue/Spoofax/152)

• Selective disable of sometimes broken completions tests in SPT. Issue and workaround described at http://yellowgrass.org/issue/Spoofax/544

• Preliminary support for building projects outside of Eclipse. Should work for most straight-forward projects that do not hook deeply into Eclipse directly. Documented at http://webdsl.org/spoofaxwiki/wiki/cmdline%20builds

- The new project wizard now supports creation of minimal project instead of full Entity - Display warning on start-up if Eclipse is not started with the necessary arguments (stack, heap, permgen) - Fixed the issue causing the file associations Eclipse preferences issue to be unusable (http://yellowgrass.org/issue/Spoofax/152) - Spoofax should be used with Eclipse Classic, as it conflicts with Web Tools Platform included in Eclipse for Java Developers. This is detailed in the issue http://yellowgrass.org/issue/Spoofax/620 and there's a note about this at http://strategoxt.org/Spoofax/Download under Incompatibilities. - Selective disable of sometimes broken completions tests in SPT. Issue and workaround described at http://yellowgrass.org/issue/Spoofax/544 - Preliminary support for building projects outside of Eclipse. Should work for most straight-forward projects that do not hook deeply into Eclipse directly. Documented at http://webdsl.org/spoofaxwiki/wiki/cmdline%20builds

Spoofax works best with a clean Eclipse Classic 3.7.2, download and install it here: http://www.eclipse.org/downloads/packages/eclipse-classic-372/indigosr2

Inside eclipse, install the unstable version of Spoofax using the following update site: http://spoofax.org/update/unstable

Documentation

Supporting material and documentation:

• A Tour of Spoofax/IMP (recommended)

• Example projects

• Test-Driven Language Development with Spoofax

• Frequently Asked Questions (FAQ)

• Editor service descriptors are documented in generated Spoofax files (e.g., YourLang-Colorer.generated.esv)

• Overview of features

• Scientific publications and presentations, including the paper The Spoofax Language Workbench, describing the architecture of Spoofax and the development of a new language and IDE

• A tutorial to SDF

• Stratego tutorial, examples, and reference manual

• The Stratego API reference

• Continuous building Spoofax plugins using Maven Tycho

• Debugging Stratego code in Spoofax Projects

• Spoofax in 14 Days (assignments from compiler construction course)

Open Issues

We maintain a bugtracker with open issues at YellowGrass.org, but if you run into any issues you can also directly contact us (see below).

• Mailing list

• Issue tracker

Community

To get in touch with the Spoofax/Stratego community use the following channels:

• The Spoofax Q&A provides a database with questions and answers

• Join the users@strategoxt.org mailing list for questions that are not appropriate for the Q&A, and for discussions about developments

• Join the IRC channel at #spoofax on freenode.net (web version). (Note that most Spoofax developers are in the CET timezone.)

Feel free to drop by!

About

The "template language" is a language that aims to eliminate redundancy in the specification of grammar, completion templates, and pretty printer. It borrows syntax from SDF and StringTemplate.

Syntax Overview

Template language sections
templates    t*	Section with templates t*
template options    o*	Section with options o*
Productions
Sort = < e* >	Template with elements e*
Sort.Constructor = < e* >	Template with elements e*
Sort = [ e* ]	Template with elements e* (alt. brackets)
Sort.Constructor = [ e* ]	Template with elements e* (alt. brackets)
Placeholders
<Sort>	Placeholder (1)
<Sort?>	Optional placeholder (0..1)
<Sort*>	Repetition (0..n)
<Sort+>	Repetition (1..n)
<Sort*; separator="\n">	Repetition with separator
<Sort; text="hi">	Placeholder with replacement text
<Sort; hide>	Placeholder hidden from completion template
Escapes
<\\\'\"\ \t\r\n>	Element containing escaped characters
<\u0065>	Unicode escape
<\\>	Escape next line break
<>	No output; layout will be allowed here in the grammar
\<</code>, \>, \[, \], \\	Escaped brackets / backslash (prefer alt. brackets)
Priority specification
context-free priorities   {left: Exp.Times Exp.Over} >   {left: Exp.Plus Exp.Minus}	Refer to templates using sort and constructor name
Lexical syntax
lexical syntax   ID = [A-Za-z] [A-Za-z0-9]*	Lexical syntax similar to SDF
Template options
keyword -/- [A-Za-z0-9]	Follow restriction for keywords
tokenize : "()"	Layout is allowed around (sequences of) these characters
newlines : none	Experimental: generate grammar that requires newline characters. Possible values: none, separating, leading, trailing

Parsing

For each syntax template there is a simple mapping to a SDF production. The semantics of the parser for the syntax template is given entirely by the generated SDF productions.

• Literal strings are tokenized on space characters (whitespace, tab). Each token results in one literal in the production. That is, hello world maps to "hello" "world", thus allowing any layout between the two tokens.
• Additionally, literal strings are tokenized on boundaries between characters from the set given by the tokenize option, and any other characters. That is, if() maps to "if" "(" Exp ")", to allow layout between the if keyword and the parentheses.
• Placeholders translate literally. If a separator option containing any non-layout characters is given, the placeholder maps to a list with separator.

Unparsing / pretty printing

Overview

The pretty printer that is generated from syntax templates consists of a number of Stratego strategies. Each matches a particular pattern in the AST. If the pattern matches, the strategy (recursively) invokes the pretty printing strategies for the child nodes, and combines the results into a BOX AST. This BOX AST can then be converted to a string using the box2text-string strategy from libstratego-gpp.

For example, for a simple plus-expression syntax template:

  Exp.Plus = <<Exp> + <Exp>> {left}

the pretty printing strategy that may be generated is:

  prettyprint-Exp:
    Plus(a, b) ->
      [ H([SOpt(HS(), "0")], [ <pp-one-Z(prettyprint-Exp)> a, S(" + "), <pp-one-Z(prettyprint-Exp)> b ]) ]

As you can see, the pretty printing strategies return a list of boxes that the caller should wrap in a V or Z box (among other things, that is what pp-one-Z does). The user is responsible for this when invoking one of the prettyprint-* strategies from their code!

All pretty printing strategies that are specific to a symbol are hooked up to one global prettyprint-LanguageName strategy, so that it is easy to pretty print an arbitrary subtree of an AST without knowing the type of the root node in advance:

  prettyprint-LanguageName = prettyprint-Exp

Parenthezation

In Spoofax, it is common to not parse parentheses to an AST node. After all, the fact that parentheses are necessary at some point in an expression can be inferred from the priorities of the operators (which are specified in the grammar) and the structure of the AST. The tool sdf2parenthesize does exactly this, and is integrated into SpoofaxLang^? via the parenthesize-LanguageName strategy. Invoking the parenthesize-LanguageName strategy results in the addition of Parenthetical constructors to the AST at every place where the structure of the AST conflicts with the operator priorities specified in the grammar. The generated pretty printer knows how to handle the Parenthetical constructor, because a syntax template for parentheses with the bracket attribute, such as:

  Exp = <(<Exp>)> {bracket}

generates the pretty printing strategy:

  prettyprint-Exp:
    Parenthetical(a) ->
      [ H([SOpt(HS(), "0")], [ S("("), <pp-one-Z(prettyprint-Exp)> a, S(")") ]) ]

How to use

The idiomatic way to invoke the generated pretty printer, including parenthezation, and a conversion to string (with word wrap at the 100th column, if applicable), is:

  prettyprint-LanguageName-string =
    parenthesize-LanguageName
    ; prettyprint-LanguageName
    ; !V([], <id>)
    ; box2text-string(|100)

-- TobiVollebregt - 01 Feb 2012

This short primer shows how to use tests as a basis for language development with Spoofax. As an example project we create a small 'calculator' language that shows many of the basics of language definitions.

The source for our example project is collected in before.zip^?. The complete project, with all functionality implemented to make the test cases succeed is collected in after.zip^?.

A full description of the testing language can be found in the paper Integrated Language Definition Testing. Enabling Test-Driven Language Development. Presentation slides also show Spoofax testing in action: http://slidesha.re/tYaHQT

A Calculator Language

Our example language supports arithmetic expressions, variables, and assignments. An example:

a = 3
a * 4

Based on this simple language we can write test cases. Doing test-driven development, these can even drive the development of the language, but in this document we focus on the tests.

Tests can be written using a .spt file. Create one in a Spoofax project, and press control-space to get a basic test definition. It will have tests of this form:

test description [[
  a = 3
  a * 4
]] 0 errors

As the example shows, each test can quote a program using [[, [[[ or [[[[ brackets. It can also specify a description for the test, and a condition. This test specifies that 0 semantic errors are expected.

Syntax

Example of tests of the syntax:

test Add [[
  1 + 2
]] parse succeeds

test Abstract syntax (1) [[
  1
]]
  parse to Int("1")

test Abstract syntax (2) [[
  1 * 2
]]
  parse to Mul(Int("1"), _)

test Parentheses [[
  (1 + 2)
]]

These tests specify parse success, compare the abstract syntax of the test input against a pattern, or simply specify that the test should succeed.

Tests can also use concrete syntax to test operator precedence and associativity:

test Multiply and add (1) [[
  1 + 2 * 3
]] parse to Add(_, Mul(_, _))

test Multiply and add (2) [[
  1 + 2 * 3
]] parse to [[
  1 + (2 * 3)
]]

test Add and multiply [[
  1 * 2 + 3
]] parse to [[
  (1 * 2) + 3
]]

test Add and add [[
  1 + 2 + 3
]] parse to [[
  (1 + 2) + 3
]]

Evaluation of expressions

The following tests will run a transformation named calc to test evaluation of expressions:

test Constant [[
  1
]] run calc to "1"

test Add [[
  1 + 1
]] run calc to "2"

test Subexpression [[
  1 + [[2 + 3]]
]] run calc to "5"

Note that the last test in this series uses the [[ ... ]] brackets to make a selection in the test. The calc transformation is evaluated for this selection.

Variables

The following tests exercise the definition of variables:

test Variable [[
  x
]] parse

test Variable [[
  longname
]] parse

test Assignment [[
  x = 4
]] parse

test Multiple statements [[
   x = 1
   y = 2
]] parse to Statements([Assign(_, _), Assign(_, _)])

    Stm*       -> Start {cons("Statements")}
    ID "=" Exp -> Stm {cons("Assign")}
    Exp        -> Stm
    ID         -> Exp {cons("Var")}

test Evaluate multiple statements [[
  1
  2
]] run calc to "2"

  calc:
     Statements(s*) -> last
     where
        s'*  := <map(calc)> s*;
        last := <last> s'*

test Eval constant [[
  PI
]] run calc to "3.14"

  calc:
    Var("PI") -> "3.14"

test Eval multiple variables [[
  x = 2
  y = x * 2 + x
  y
]] run calc to "6"

Editor features

The following test cases test the editor facilities of the language:

test Variable unassigned [[
  y
]] 1 error /unassigned/

This test succeeds if the input has 1 error and it matches the regular expression /unassigned/ (as variable y is unassigned).

test Multiple assignments to same variable [[
  y = 1
  y = 2
  y
]] /multiple/

This test succeeds if there are one or more (error) messages that match /multiple/ (as there are multiple definitions of y).

test Reference resolving (1) [[
  x = 4
  [[x]]
]] resolve

test Reference resolving (2) [[
  [[x]] = 4
  [[x]]
]] resolve #2 to #1

test Content completion [[
  avariable = 1
  [[a]]
]] complete to "avariable"

These test cases test reference resolving and content completion. They use the [[ ... ]] selection mechanic to select parts of the program. The first test case specifies that reference resolving should work for the selection. The second specifies that the second selection should resolve to the first selection. The last test case specifies that the selection should provide a content completion option avariable.

Execution

The Calculang project generates Java code. The following test case tests the output that is returned when the program is compiled and executed:

test 42 [[
  42
]] build generate-result to 42

This page provides a tour of the different features of Spoofax. For additional supporting material and documentation, visit our Documentation page. For a quick overview of features, see the Features page.

Tip: We provide general Spoofax usage tips using boxes like this one throughout this tour.

Installation

Moved to http://metaborg.org/wiki/spoofax/download

Getting Started

To get started, create a new Spoofax/IMP project using the File > New > Project > Spoofax/IMP project wizard.

The wizard creates a new an example language and editor definition. It also opens editors for the crucial parts of these definitions, as illustrated below:

Highlighted in blue are components that can be used to customize the language and editor. Key components (show in tabs, from left to right) are the SDF syntax of the language, the main editor descriptor, and the Stratego semantic description. The wizard also opens an editor with an example file that can be used to try out the language editor.

We recommend you glance the files in the project to get an idea of what function they have. The wizard filled in some default content that can be edited. Other files are derived files: in particular, all files with .generated in their name are derived from the language definition. These files should not be edited directly as they are generated every time the project is built.

Tip: To rebuild the language and editor at any point use Project > Build Project (control-alt-B) or right-click on the build.main.xml file and run it using Ant.

The syntax definition

The SDF grammar in the syntax directory defines the syntactic structure of the language:

The easiest way to work with the syntax may be to look at it side-by-side with an example editor, allowing you to experiment with the grammar and a concrete file.

Tip: To show two editors side-by-side in Eclipse, drag their caption to a position adjacent to another editor.

The syntax of the default entities language has four production rules: a start production with the module keyword, and productions for entities, properties, and types. SDF productions take the following format:

p* -> S {a*}

And specify that a pattern p* reduces to a symbol S (sometimes called a sort). Using annotations a*, the abstract syntax and exact behavior of the production may be customized. In particular, we use the cons("Name") to specify that "Name" should be used to label the abstract syntax tree node. The abstract syntax is relevant for code generation and analysis of the language. We revisit this topic further on in this section.

To best get a feel of the SDF grammar formalism, try adding a new production to the language. For example, add a new kind of entity property that has a different (composite) relation to its type:

ID "<>" Type -> Property {cons("CompositeProperty")}

Note how SDF just allows you to add an additional case to the Property symbol. SDF can be used to modularly reuse and extend existing grammars in this fashion. The constructor label "CompositeProperty" is used to uniquely identify this new property relation. To try it out, rebuild your project (using control-alt-B), and use it in the example editor.

Tip: For more information on using the SDF, see the reference manual on the Documentation page.

The abstract syntax

Spoofax can show the abstract syntax of your program as textual terms. These are efficiently stored in memory and allow you to concisely manipulate programs. To inspect the abstract syntax of a file, select Show abstract syntax from the Transform menu:

An extra editor will be opened to the side showing the abstract syntax of your file or selection. Note that the view is updated as the original file is edited.

Presentational editor services

Presentational editor services such as code folding, the outline view, and syntax highlighting are defined using esv files in the editor directory.

Code folding

As an example, we take a look at the definition for code folding:

The default EntityLang-Folding.generated.esv file defines folding for the start production (symbol Start, constructor label Module) and for entity definitions.

All editor services follow the same basic structure: they have a definition file (EntityLang-Folding.esv) and a derived file (EntityLang-Folding.generated.esv). The first file can be used to customize the editor. The second file is automatically generated from your grammar, each time the project is built.

Tip: The .generated files document the editor descriptor languages using comments.

Outline view

The outline view of the editor can be customized in the same way as the folding definition:

An interesting feature of the outline is that it can be shown in the quick outline view by pressing control-O (command-O on Mac OSX). This allows users to quickly navigate in the file:

Syntax highlighting

We can use the EntityLang-Colorer file to customize the generated syntax highlighter:

Transformations and code generation

The generate.str file in the trans directory defines code generation rules:

These generate Java code directly from the abstract syntax of the language. Each takes the following form:

name:
  T1 -> T2
  with
    x := y;
    ...

A rule of this form rewrites a subtree of form T1 to T2. In the optional with clause of the rule, variables can be assigned and other rules can be invoked. Some rules specify a where clause with a condition for the rewrite rule.

The rules in our example directly rewrite to Java code using string interpolation to construct Java as text fragments. String interpolation allows you to combine text with variables. Any indentation you use is preserved in the end result, minus the indentation leading up to the quotation. This means your code looks pretty both when you write it and when it comes out of the generator. You can use string interpolation using the $[ ... ] brackets. Variables can be inserted using brackets without a dollar: [ ... ]. If you want, you can also use any other kind of bracket: {, <, and ( are allowed.

You can try out the transformation using the Transform menu, selecting Generate Java code:

If you edit the input text, the transformation result will be automatically updated. Try changing the transformation and rebuilding the project.

Advanced transformations

When you implement more complicated transformations you will find that it is easier to use intermediate rewrite rules instead that rewrite entity ASTs to simpler entity ASTs (sometimes called model transformations or desugarings). To accomplish this, you can add additional definitions for the to-java rule, or you can add a new rule with a different name:

This rule rewrites the CompositeProperty we have shown before to a regular property. We can now rebuild the project and try out the transformation. Select a composite entity property in the editor or abstract syntax view and choose Apply custom rule in the Transform menu:

When prompted, enter the name of the simplify rule:

The result of our transformation is now shown in an a view to the side:

To use the simplify rule in our transformation to Java, we must make sure that it is invoked there. The original to-java rule was invoked in entitylang.str, which defines the main interface of our language for the editor:

This rule follows a fixed interface for interoperability with the editor, and gets the node currently selected by the user, its tree position as a list of indices, the complete ast of the file, and the path of the file. As a result, it returns a tuple of a filename that should be generated and its contents.

We can adapt the generate-java rule to call our new simplify rule:

We added a new rule to-java-top that first invokes the simplify rule before to-java is invoked. However, when you rebuild the project you will find that this adaptation does not work: the simplify rule cannot be applied to an entire module. Unlike to-java, it is only defined for CompositeProperty.

We can fix our transformation by introducing new definitions for the simplify rule for modules, entities, etc:

Unfortunately, this practice quickly becomes tedious and hard to maintain. Stratego also allows us to specify a strategy according to which the rule should be applied in the entire tree. A common strategy is the topdown/try strategy: walk the tree in a topdown fashion, and try to apply it everywhere in the tree where possible:

Other useful strategies are map, to apply a rule on a list (e.g., map(simplify)), and innermost, to exhaustively apply a rule, starting at the innermost leaves of a tree.

Tip: For more information on using strategies and rewrite rules in Stratego, see the reference manual and tutorial on the Documentation page.

Customizing the Transform menu

The Transform menu allows you to apply any rule to a file or selection:

Transform can be used directly in an editor of your language, but also from the abstract syntax of that language.

The EntityLang-Builders.esv file in the editor directory defines the entries in the Transform menu:

The corresponding EntityLang-Builders.generated.esv documents the definitions in this file. For this tour, we are interested in the builder definitions, which control the Transform menu items. The first builder specifies the "Generate Java code" item. It indicates that the generate-java rule that we saw before is responsible for this action. Using annotations, it also specifies that this action opens an editor which will be updated in "real time" as the user edits the source of the tranformation. The second builder is a meta builder: it only shows up to developers of the language, not when the plugin is deployed.

Error checking

Any errors in the entities language are reported directly in the editor:

These errors are specified using check rules in the check.str file in the trans directory:

Like code generation rules, these rules are really just regular rewrite rules. They match against a particular language element and produce a tuple of the offending language element (where the error marker will be shown) and the error message.

The first definition of constraint-error matches properties where a particular condition holds (note the use of the where keyword): the property type is not "String" or "Int", and using GetEntity (defined below) we can tell that it does not exist in the symbol table.

Refactorings

Spoofax helps you to enrich your editor with refactorings. You can find the refactorings in the context menu of the file being edited.

Refactoring Specifications

The EntityLang-Builders.esv file in the editor directory defines the entries in the Refactor menu:

The given refactoring specifies the "Rename" menu item, which is enabled in case an ID node is selected.The menu action is bind to the rename shortcut (Shift+Alt+R). After the user applies the rename refactoring, a dialog is prompted with an input field labeled "New name" that has the empty string as initial value.

The OK button of the dialog triggers the action defined with the rename rule that we discuss in the next paragraph.

Refactoring Transformations

Refactorings are defined in Stratego. The refactor.str file in the trans directory defines the renaming refactoring for the entity language.

This rule follows a fixed interface for interoperability with the editor. The left-hand-side of the rule is a tuple of: the result of the user input dialog, the selected node, its tree position, the complete ast of the file, the file path and the project path. The right-hand-side is a tuple containing the refactoring output: a list of node changes, plus lists with errors and warnings that will be reported to the user. Errors and warnings are specified as a tuple of the offending language element (which location will be reported) and the error message itself.

Tip: multiple-file refactorings can be specified in the change list using the root nodes: [(ast-1-before, ast-1-after), (ast-2-before, ast-2-after), ...]

Program analysis

We use a symbol table as the basis for our semantic analysis. Every entity is stored in this table, mapping its name to its definition. This table is defined in the record-entity rule in check.str:

The analyze rule uses the topdown/try strategy we have shown before. It will try to apply the record-entity rule wherever it can. The record-entity rule is applied to all entities: User, BlogPosting, and URL. Stratego provides a concise notation for creating symbol tables: using the rules keyword, we define a new entry for the name 'x' that points to the node in the abstract syntax tree that defines it. We use this definition for error checking, reference resolving, and content completion.

Reference resolving

When a user holds control (or command on Mac OSX) and hovers over an identifier, they get a link to its definition:

Supporting reference resolving is easy once a program analysis is performed. By default, the entities language simply invokes the GetEntity rule to fetch the definition for any Type selected by the user:

This rule follows the same structure as other rules in this file. It first prints a debugging message "Resolve: ...", which shows up in the Eclipse console (but not when the plugin is deployed). It then tries to set a variable type using the node that was just selected by the user. If the node is of the form Type(_), then it will call the GetEntity rule to resolve the declaration of that type. Since editor-resolve uses a where clause, it will only succeed if node is in fact a Type and if GetEntity succeeds. If this is not the case, another definition may of editor-resolve may be used instead, if one exists.

Content completion

Spoofax supports content completion that can be triggered by pressing control-space:

Content completion can also be triggered when the user enters a particular sequence of characters (like the . in Java). The EntitityLang-Completions.esv allows this behavior to be configured:

This file specifies that content completion should be triggered once users enter ": ". It also says that the editor-complete rule handles the implementation of content completion.

Like other editor services in Spoofax, content completion is implemented at the abstract syntax level. Once content completion is triggered, an AST like the following is created:

In this AST we see a special COMPLETION(_) note where content completion is requested. The editor-complete rule in entity-lang.str provides completion proposals:

This rule gets the current abstract syntax tree node, its position, etc. and returns a list of content proposals. The "String" and "Int" proposal are hardcoded; other proposals are computed by taking all the keys of our GetEntity table. For this we use the all-keys-GetEntity rule which is automatically made available from our GetEntity definition. Spoofax will then automatically show proposals with the right prefix (here the letter "U").

Testing Language Definitions

Spoofax allows you to write tests for your language implementation using a specialized testing language. An example set of tests can be found in test/test-example.spt.

A basic testing module has the following form, specifying the name of a language:

It specifies the name of the language under test (EntityLang) and one or more test cases. The test case in this module is based on a small EntityLang program enclosed in ^? brackets, and specifies that this program should parse to a pattern Module("example", [Entity("User", _)]).

More advanced test cases may use setup blocks to factor out commonalities in test cases, and can specify a wide range of testing conditions:

These test cases test for semantic errors (0 errors or 1 error) and test that refactoring works as expected. Notice how the last test case uses the ^? brackets to indicate that a rename refactoring is performed on the "User" entity. This test case simulates the renaming of the "User" entity to a "Customer" entity.

Any failing test cases are marked with an error in the editor. The "Transform" menu can also be used to launch a project-wide, graphical test runner:

Tip: content completion shows more conditions that can be specified for test cases. A full overview is also given in the paper Integrated Language Definition Testing.

Plugin deployment

You can deploy your plugin using the Eclipse update site mechanism. Users do not need to have Spoofax/IMP installed for using your plugin. To create an update site, you need to create a plugin feature project and an update site project.

You can create a new plugin feature project by using the File > New > Project > Plug-in development > Feature Project wizard:

In the tab plugins, add your plugin to the list of included features using the "Add" button.

In the tab Included Features you should add Spoofax/IMP. Be sure you select that it can be updated from "both" update sites for maximal compatibility with existing installations and updates:

You can then create an update site project. From Overview page of feature.xml, open the New Update Site wizard:

As the name of the update site you may use company.entities.updatesite. In the update site project, create a new category and add the feature project:

From here, you can select Build All to build the update site. You can then upload it to a web server and use it to install or update your plugin using the Eclipse update manager.

Adding Java components

Some advanced users may want to add their own Java components to their plugins. Spoofax provides two mechanisms for interoperation with Java and other JVM-based languages. The first mechanism is the standard Eclipse extension mechanism: as the Spoofax framework is based on Java and Eclipse, it allows you to add new, Java-based components as Eclipse extensions. The other mechanism is that it allows users to implement their own strategies (functions that follow the Stratego interface) using Java. This mechanism is documented in the editor/java/.../strategies/java_strategy_0_0.java example strategy included with the default project. Strategies implemented this way can be dynamically loaded just like other Spoofax components.

In this tour we focus on the Eclipse extension mechanism. These components can be added using the Extensions tab of plugin.xml:

You can then select Add to add a new extension:

Note that only editor components defined using Spoofax/IMP can be used directly in the same Eclipse instance in which they are developed. Normal Eclipse plugins and extensions do not have this feature. You can try your Java-based extensions using the debug or run buttons in the tool bar, launching a new Eclipse instance:

A key class for integrating with Spoofax editors from Java org.strategoxt.imp.runtime.EditorState class, which provides things like the current or selection AST of an open editor. It also allows model listeners to be attached using EditorState.getEditor().addModelListener(). Please contact us if you have any questions about Java integration or other topics; we'd be happy to help.

All built-in Spoofax/IMP editor components are based on the Eclipse IMP framework. In addition to standard Eclipse plugin extensions, IMP can also be used to define parts of your editor. These can be created using the File > New wizard, selecting any of the IDE Language Support options. At this time, however, not all these wizards are stable, and they may not always work right within the Spoofax/IMP environment. Instead, we recommend that users build editor services using the Spoofax-native esv definitions or, if necessary, using regular Eclipse extension points.

Topic	Changed	By
WebLeftBar	11 May 2017 - 05:48	EelcoVisser
CommandLine?	12 Jun 2014 - 09:03	VladVergu
Testing?	13 Sep 2013 - 14:42	EelcoVisser
News?	25 Mar 2013 - 14:42	LennartKats
NBL?	25 Mar 2013 - 10:57	EelcoVisser
Download?	25 Mar 2013 - 10:56	EelcoVisser
ChangeLog?	22 Mar 2013 - 10:40	LennartKats
SpoofaxRelease11?	16 Feb 2013 - 16:40	EelcoVisser
NaBL?	14 Feb 2013 - 20:33	EelcoVisser
DebugStratego?	01 Feb 2013 - 09:14	GuidoWachsmuth
SemanticIndex?	01 Feb 2013 - 09:12	GuidoWachsmuth
ContinuousBuildingSpoofax?	01 Feb 2013 - 09:11	GuidoWachsmuth
Development?	28 Jan 2013 - 13:56	EelcoVisser
Support?	28 Jan 2013 - 13:06	EelcoVisser
Documentation?	22 Oct 2012 - 22:10	EelcoVisser
FAQ?	17 Oct 2012 - 12:28	LennartKats
SummerSchool?	23 Jul 2012 - 06:31	GabrielKonat
TemplateLanguage?	09 Apr 2012 - 04:45	TobiVollebregt
SpoofaxLanguage?	24 Mar 2012 - 14:30	TobiVollebregt
WebHome	24 Feb 2012 - 17:28	LennartKats
Refactorings?	26 Jan 2012 - 13:42	MaartjeDeJonge
ExampleIni?	21 Dec 2011 - 15:16	LennartKats
TestingDemo?	20 Dec 2011 - 14:51	LennartKats
Research?	18 Dec 2011 - 08:20	MaartjeDeJonge
Tour?	28 Nov 2011 - 14:37	LennartKats
Features?	27 Nov 2011 - 00:57	MaartjeDeJonge
Examples?	11 Oct 2011 - 16:01	TobiVollebregt
LWC2011?	24 May 2011 - 08:13	LennartKats
LayoutPreservation?	19 Nov 2010 - 21:45	EelcoVisser
Devnology?	06 Nov 2010 - 14:43	LennartKats
InteractiveDisambiguation?	27 Jun 2010 - 22:22	LennartKats
NWL?	06 May 2010 - 11:10	LennartKats
Components?	31 Mar 2010 - 15:02	LennartKats
IMP?	23 Feb 2010 - 10:23	LennartKats
WebPreferences	12 Feb 2010 - 14:50	LennartKats
WebRss	16 Aug 2004 - 03:27	PeterThoeny
WebSearchAdvanced	18 Jan 2004 - 10:52	PeterThoeny
UpdateWebPages	20 Sep 2002 - 08:37	EelcoVisser
SiteMap	27 Aug 2002 - 08:00	EelcoVisser
WebChanges500	23 Apr 2002 - 20:09	EelcoVisser
WebChanges200	23 Apr 2002 - 20:05	EelcoVisser
WebChanges100	23 Apr 2002 - 19:53	EelcoVisser
WebChanges	23 Apr 2002 - 19:46	EelcoVisser
WebNotify	23 Jan 2002 - 14:21	EelcoVisser
WebIndex	23 Jan 2002 - 14:20	EelcoVisser
WebNews	23 Jan 2002 - 14:17	EelcoVisser
WebTopicList	24 Nov 2001 - 11:40	PeterThoeny
WebTools	08 Nov 2001 - 09:49	TWikiGuest
WebSearch	08 Aug 2001 - 05:26	PeterThoeny
WebStatistics	08 Aug 2001 - 05:25	PeterThoeny

See also

• WebChanges100
• WebChanges200
• WebChanges500

Spoofax is a platform for developing textual domain-specific languages with full-featured Eclipse editor plugins.

With the Spoofax/IMP language workbench, you can write the grammar of your language using the high-level SDF grammar formalism. Based on this grammar, basic editor services such as syntax highlighting and code folding are automatically provided. Using high-level descriptor languages, these services can be customized. More sophisticated services such as error marking and content completion can be specified using rewrite rules in the Stratego language.

News